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ABSTRACT 



This conference paper discusses the results of a study that 
investigated the effectiveness of the Total Family Strategy (TFS ) , a 
home-based family support program funded by the Juvenile Welfare Board of 
Pinellas County, Florida. The program was initiated as a means of introducing 
a family empowerment model of early intervention into the array of existing 
services for children and families. The target population was young families 
with children under the age of 12 who are at risk of developing serious 
problems and who have not already utilized significant social service 
resources. Four agencies in the community provide this service. Program 
components include case management, access to community resources, advocacy, 
and counseling. The evaluation study included data from agency directors and 
93 families and intensive case studies with eight TFS families. Results 
indicated: (1) agency directors had different ideas about the purpose of the 

program, its target population, and indicators of the program effectiveness; 
(2) many families being served had already experienced significant systems 
involvement; and (3) agencies appeared to interpret program guidelines 
differently, leading to differences in program implementation. (CR) 
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Introduction 



Family support programs have been implemented across a 
wide variety of service systems for children and families 
(Ayoub, Daro, Landsverk, & Lutzker, 1996). Development 
of effective knowledge about "what works" in these 
programs has been hampered by a number of program 
implementation and evaluation issues (Pecora, Fraser, 
Nelson, McCroskey, & Meezan, 1995). These include lack 
of a coherent theory of family-based services, poorly 
specified program models, inconsistent referral and case 
targeting practices, absence of procedural guidelines for 
direct service issues (e.g., staff selection, training, and 
retention), and an inability to develop a systematic 
mechanism for monitoring program quality and refining the 
program as it stabilizes and matures. 

In this summary, a local family support program is 
presented in order to illustrate these issues and highlight the 
importance of appropriate planning, monitoring, and 
evaluation of family support efforts. The Pecora et al. 
(1995) framework is also expanded by describing the 
significance of contextual factors which affect program 
implementation and evaluation; such a description has 
implications for third-party funders of services, providers, 
and evaluators. 
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Method 



Total Family Strategy (TFS) is a home-based family support 
| program funded by the Juvenile Welfare Board (JWB) of 
j Pinellas County, Florida. JWB is an independent special 
taxing district that funds delivery of children's services 
through other community agencies (Juvenile Welfare Board, 
j 1992). 

This program was initiated by JWB as a means of 
| introducing a family empowerment model of early 
intervention into the array of existing services for children 
and families. The goal of TFS is to prevent "million-dollar 
families" (Friedman, 1991), who could incur that amount in 
service system costs throughout the life of the family. The 
target population is young families with children under the 
age of 12 who are at risk of developing serious problems, 

1 and who have not already utilized significant social service 
| resources. Four agencies in the community provide this 
I service. Program components include case management, 

! access to community resources, advocacy, and counseling, 
j The University of South Florida, Florida Mental Health 
Institute (FMHI) has been involved from the inception of 
the program in conceptualizing the program and evaluating 
its impact. j 

Beginning in 1 994, a process evaluation of the TFS program ! 
was conducted (Silver & Marfo, 1994). Multiple methods 
were used in the TFS evaluation, including (a) review of ! 

existing program descriptions and other documents, (b) 
interviews with agency directors and staff, and (c) 
quantitative analysis of data routinely collected from 
families (N — 93). Intensive case studies were also 
conducted with TFS families (N = 8) and staff. 

| j 

j Results 

| Process evaluation revealed a number of issues identified by I 
: Pecora et al. (1995) operating in the TFS program, as 
follows. J 

Under-specification of the program model. Interviews 
conducted with agency directors revealed different ideas 
about the purpose of the program, its target population, and 
indicators of program effectiveness. For example, responses 
to questions about the purpose of the program revealed a 
range of program philosophies, from a focus on preventing 
serious problems, such as out-of-home placements or 
juvenile delinquency, to an emphasis on empowering 
families to problem-solve, access resources, develop 
informal networks, and become more self-sufficient. These 
differences in program philosophy also occurred at the 
program staff level, as revealed by case study interviews 
and observations at monthly staff meetings. 

Inappropriate referral and case targeting. Case studies and 
quantitative analysis indicated that many families being 
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quantitative analysis indicated that many families being 
served in the program had already experienced significant 
systems involvement. Examples from the case studies 
included involvement in the child protective services 
system, adult corrections, and severe emotional difficulties. 
This population differed significantly from the one that the 
program was designed to serve; moreover, the population 
which was served varied by agency (Silver, 1 995). Historic 
differences in populations served, referral sources, and 
funding sources may have contributed to different 
interpretations of the eligibility guidelines. 

Program drift. The four agencies appeared to have a similar 
picture of the program model when the initiative began, as 
evidenced by a set of guidelines they had jointly developed. 
However, agencies appeared to interpret the guidelines 
differently, which led to differences in program 
implementation. For example, as a result of differences in 
agencies' staffing patterns, differences occurred in the 
degree to which therapeutic services, versus case 
management and referral, were emphasized in staffs work 
with families. Record-keeping procedures were modified to 
meet requirements of the larger agency, which sometimes 
affected methods used in family assessment and case plan 
development. Staff turn-over, combined with lack of 
detailed implementation procedures, increased the 
opportunity for differences in the interpretation of the 
program model. 

Discussion 

Process evaluation identified significant variation in 
agencies' implementation of this family support program. 

As a result of this and other information, program staff, 
JWB, and the evaluator are developing mechanisms to 
improve implementation on the issues described above. 

There were important contextual issues in the current 
initiative that shaped program implementation, monitoring, 
and evaluation activities. The first issue pertains to the 
relationship between funders and service providers. In 1991, 
JWB and FMHI identified the need for an innovative 
service and sought to have this service provided by local 
agencies. The impetus for the initiative came from outside 
the agencies. A second, related contextual issue is that the 
model underlying TFS represented a change in agencies' 
existing practices. Total Family Strategy emphasizes a 
home-based (versus office-based) model, working with 
families holistically instead of with the identified client. 
Additionally, TFS focused on partnering with families, 
building on their strengths and providing community-level 
supports, as opposed to employing the traditional 
therapeutic approach. Despite the joint development of 
program guidelines, the process evaluation indicated that 
agencies were operating with different interpretations of 
how this model translated into practice. This 
underspecification of the program model led to uneven 
program implementation across agencies. 
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Contextual factors also affected quality control and program 
evaluation activities. Initially, neither JWB nor FMHI fully 
realized the complexity of this initiative, including the time 
needed to implement and bring about the desired changes. 
As the program developed, recognition of the necessity for 
developing new approaches to contract management and 
I evaluation also occurred. "Lessons learned" included 
changing from a practice of separate JWB contract 
| managers (i.e., one for each agency) to one contract 
| manager who worked across all four agencies and assumed 
! a greater coordinating role. 

Pecora et al. (1995) note that one of the pitfalls of 
evaluating programs is a mismatch between stage of 
program development and evaluation design. The initial 
| evaluation design focused on short-term outcomes (i.e., an 
| approach consistent with JWB's practice of monitoring 
| contracts with established programs). As the complexity of 
j managing this program revealed itself, JWB requested an 
! evaluation plan that included both process and outcome 
| components. Process evaluation revealed a number of issues 
] that will impact long-term evaluation of this program (e.g., 
j the differences in the populations served by agencies will 
• require analysis of families’ long-term progress separately 
j for each agency). This has led to a clearer understanding of 
| the limitations of the evaluation that would not have been 
j achieved without a focus on process evaluation. 

j 

j The nature of family-support programs affects the context, 
j implementation, and evaluation of programs. Family 
| support services offer a wide breadth of services, and issues 
j faced by families represent multiple domains (e.g., housing, 

! basic needs, transportation, additional education/training, 

; parenting support, and counseling). The heterogeneity of the 
; target population poses significant challenges for family 
! support programs. In the current program, these challenges 
| may have been compounded by the difficulty of 
| operationalizing characteristics of those likely to become 
| "million dollar families." In addition to significant process 
evaluation issues, family support programs present 
formidable challenges to outcome evaluation. Evaluation is 
made more difficult by the use of individualized criteria for 
determining family progress and success, and by the fact 
that family success is also dependent upon available and 
accessible community resources&endash;issues which are 
often not under the control of program staff (Silver, 1995). 

Experience with the TFS evaluation leads to several 
recommendations. These recommendations apply to 
funders, providers, and evaluators of family-based 
programs. The importance of careful, systematic 
implementation planning cannot be overlooked. This starts 
with review of Pecora, et al.'s (1995) description of the 
issues involved in implementing and evaluating services. It 
is also crucial to allocate sufficient time and resources to 
monitor program implementation. The specific strategy used 
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to monitor implementation can take a number of forms. 

These include having the funder and providers develop 
self-assessment mechanisms that would include 
"benchmarks of progress" to determine whether program 
implementation is on track (e.g., Is there a mechanism for 
regular review and refinements of the program model? What | 
information is being collected to provide feedback about 
these issues?). If the time needed to develop, implement, 
and monitor the program exceeds that of the funder and/or 
providers, consider having an external evaluator conduct a 
process evaluation. A third strategy combines elements of 
the first two (e.g., having an evaluator provide technical 
assistance on developing self-assessment measures). 

Carefully consider the context in which this new program is 
being implemented. Is this an innovative service that 
challenges traditional practices? Who is in charge of the 
innovation? How does this model fit with existing 
philosophy, practice, and training of providers? What group 
processes can be used to facilitate the empowerment of 
providers and their ownership of innovation? Be sure to 
define roles, including the role of the evaluator. Consider 
how the role of an evaluator can be utilized to help develop 
a more participatory approach to program evaluation (e.g., 
as a participant observer who provides technical assistance 
and helps facilitate the group process, vs. a traditional 
evaluator that may lead providers to comply, rather than 
take risks and try an innovation). 

The process issues described in this summary raise 
questions about how providers and funders can work 
together to develop and deliver innovative family support 
programs. Developing mechanisms to foster collaboration 
may be especially important when the new initiative 
requires changes in existing funder-provider relationships 
and staff practices. These issues are likely to grow in 
importance as family support programs grow and pressures 
for greater accountability of programs increase. 
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